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ABSTRACT 

The Multimedia Forum Kiosk (MFK) is an on-line 
discussion tool that has been tested as a new technique for assessing 
curricular reform. MFK collects discussion comments entered into the 
computer by students and instructors. This paper explores experiences 
with the MFK as a technique for on-line assessment using multimedia. 
The context of the assessment is an eight-university coalition of 
engineering schools called Synthesis, which is sponsored by the 
National Science Foundation. In this study, the MFK assessment 
approach was used to document community-wide reactions to aspects of 
curricular innovation, to foster discussion about new pedagogical 
practices, and to sample student skills and attitudes that could be 
demonstrated in an engineering discussion. The MFK allows discussion 
in an opinion area allowing for one statement of opinion and a 
discussion area that allows for exchanges of comments. Face icons of 
all the participants in the discussion made their identities salient 
and gave discussions a more personal tone. In this context, the MFK 
was very fruitful in examining curricular reform, a situation in 
which goals were vague and not based on declarative knowledge. By 
involving the instructors in conversations with their students, the 
MFK facilitated the evaluation of curriculum impact. Capturing 
discussions is a viable way to examine processes and attitudes and a 
new tool for open-ended assessment. (Contains 1 table, 1 graph, and 
10 references.) (SLD) 
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The Multimedia Forum Kiosk (MFK) [Berman, Hsi, & Hoadley, 1994; 
Berman, Hsi, Hoadley, & Linn, 1995; Hoadley & Hsi, 1992] is an on-line 
discussion tool that has been tested as a new technique for assessing curricular 
reform. MFK collects discussion comments entered into the computer by 
students and instructors. Preliminary data suggests the multimedia interface 
of MFK encourages productive discussion and reflective comments [Hoadley 
& Hsi, CHI 1993] and can be effective for encouraging group communication, 
full participation, and collaborative knowledge building [Hsi & Hoadley, 
AERA 1994], 



This paper shares our experiences with the Multimedia Forum Kiosk as a 
new technique for on-line assessment using multimedia. We focus on three 
key aspects of the Kiosk, and we present examples from engineering to 
demonstrate how the tool supports group assessment and capitalizes on the 
multimedia nature of the electronic medium. 



Motivation: Assessment of Curricula 




The context of our assessment is an eight university coalition of engineering 
schools called Synthesis. Sponsored by the National Science Foundation, the 
coalition's goals are to improve undergraduate engineering education and to 
better prepare engineering students for the workplace. This is being achieved 
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through modifying existing curricula, developing multimedia courseware, 
testing new instructional methods, and inventing new technology-based 
solutions to instruction. [NSF EEC, 1990]. 

Assessing curricular reform poses interesting challenges. First, instructors in 
the process of developing and refining educational materials to meet general 
pedagogical goals have difficulty measuring the impact of their changes. 
Oftentimes, a problem in the curriculum or in student outcomes is perceived, 
but it is not always clear what would be better, or even what the criteria of 
improvement would be. Instructors sometimes don't know in what direction 
their curriculum should be changed. 

Second, a challenge in assessing curricular reform is that the purpose of 
reform often addresses both global and local concerns. In our case, global 
concerns included producing engineers who are better at communicating, 
synthesizing information, and negotiating in the marketplace. These 
concerns must be kept in mind while local objectives such as teaching 
students a specific engineering analytic approach are also being met. An 
assessment method needs to be flexible to capture students' reactions to both 
local and global aspects of changes to the curricula and to instruction. 

Thirdly, curricular reform often focuses on non-declarative aspects of 
learning. For instance, one goal of the Synthesis Coalition is higher retention 
of women and ethnic minorities in engineering. Unlike traditional content 
areas, assessing the curricular impact on retention involves a complex 
interplay of social mores, attitudes, and long-term behavior, none of which is 
easily captured in a course test. Another example would be Synthesis' 
emphasis on skills for teamwork. By nature, these skills are not demonstrable 
in a traditional hour-long written test. 

To address these needs, the Multimedia Forum Kiosk assessment approach 
had the following objectives: (a) to document community-wide reactions to 
aspects of curricular innovations, (b) to foster dialogue and discussion about 
new pedagogical practices, and (c) to sample student skills and attitudes which 
could be demonstrated in an engineering discussion (such as a design 
discussion). In doing so, the system would complement more outcome-based 
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measures of curricular success, such as test scores or employment rates. The 
system filled three important gaps left by traditional classroom testing: 
collection of student opinions of the new instruction, an appropriate way for 
students to contribute to the curricular evolution, and a way of gauging how 
students interact with each other. 

Brief Description of the System 

The Multimedia Forum Kiosk allows two types of discussion in two different 
areas: The Opinion Area and the Discussion Area. (Figure 1 and Figure 2) In 
the Opinion Area, students type in their overall reaction to a topic phrased as 
a question. Each topic has an author, typically the instructor, who introduces 
the topic with a text overview. Users can read the overview provided by the 
instructor, watch digital movies, read about the stances of others, or post their 
own viewpoint. For instance, a topic might be "Which of these experiences 
has helped you become a better engineer?" and could be accompanied by 
images of activities from the course, links to other documents such as 
homework assignments, or video clips of a student discussing an 
extracurricular experience. Each comment is represented by a face icon, so the 
identity of the contributor is immediately obvious; on the other hand, 
students may remain anonymous and choose a cartoon icon if they wish not 
to be identified (if they are being critical of the instructor, for example). Each 
person may give only one opinion in the Opinion Area, much like voting on 
an issue. These responses are open-ended, however, and may be edited at a 
later date to reflect changes in the student's stance. Open-ended responses 
allow students to decide what aspects of the curriculum to mention and to 
explain their answers in as much detail as necessary. Students may glance at 
the screen and see who is participating in the discussion, then discover their 
opinion on the topic at the click of a mouse. As a collection, the comments in 
the Opinion Area serve as the community perspective on the topic theme. 

The second representation of discussion is called the Discussion Area. The 
Discussion Area allows users to make multiple comments, and to respond to 
the comments of others. This allows an ongoing, back-and-forth type of 
discussion. Each comment in the Discussion Area is categorized according to 
the semantic labels (AND, BUT, OR, I.E., ?, ...) and linked to a previous 
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comment to form argument trees. Students may respond at any time to any 
comment made in the past. This representation lends itself to following lines 
of reasoning, tracing previous discussions, and (for the researcher performing 
assessment) analyzing comments. 

The Multimedia Forum Kiosk is currently implemented in HyperCard 2.2 
with QuickTime for the Apple Macintosh. It includes an authoring toolkit 
which allows instructors to design topics easily, (see Figure 3) Currently we 
are reimplementing the system to allow networked access via the World 
Wide Web. 



How was MFK used? A Description of an Evaluation 

Identifying Themes 

The first step in the MFK assessment approach was to meet with instructors 
to identify central themes to investigate. In the context of our assessment of 
engineering education, we arrived at four themes chosen to also reflect the 
Synthesis objectives: (1) relevance to engineering practice, (2) Synthesis 
learning experiences, (3) classroom learning atmosphere, and (4) design 
learning and instruction (Table 1). These themes not only structured the 
discussion topics used for evaluation, but also helped to organize the data 
analyses. 



Identifying Topics 



For each course, several topics were designed by instructors with the help of 
evaluators that were consistent with the four assessment themes. For 
instance, some sample topics are presented here from two mechanical 
engineering courses. 

What are some effective ways to teach design? 

Case-base advocates of learning believe that students learn by studying many 
examples of good design. Others believe in the brute force approach where 
students gain experience through building as many prototypes as possible in the 
studio or shop. Do you agree with these viewpoints? What are other methods 
you have used to learn engineering design? 
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Is your classroom or lab a supportive environment for learning? 

The classroom, like a design environment, can greatly affect a student's ability 
to learn. It is important to provide an environment that fosters learning and 
exploration. The instructor and students should be both listeners and active 
participants. Do you feel this is the case for ME290P? 

What do you think you are learning in ME107A? 

In ME107A, we try to teach you general principles of experimentation that 
every engineer should know. What do you think you are learning in 107 A 
versus what you think we are trying to teach you? 

ME107A: Guided learning or trial and error? 

Are the 107 A labs more like guided learning or like trial and error? As 
instructors, we want to encourage you to use your engineering judgment in 
planning an experiment. At the same time, we know this is the first time you've 
seen this hardware. Would you like more detailed instructions on lab 
procedures, or do you like the freedom to plan your own experiment? 



Students were given from one week to one 15 week semester to respond to a 
set of topics. Typically, an MFK was set up in a location where students could 
access the conversation at their convenience, such as a course laboratory. 
Students who want to voice opinions about new curricular materials, the 
learning atmosphere, or instructional delivery could interact with the Kiosk 
during the semester either voluntarily or as an assigned independent activity. 

During the course evaluation period, comments and discussion were 
automatically logged by the computer. Instructors could read the comments 
interactively, or print out hard copies to keep a record of student feedback. 



Why Group Assessment? 

A feature of the Multimedia Forum Kiosk assessment approach is its ability to 
capture comments collaboratively and share them in a group. Unlike surveys 
that are collected at the end of the semester and read by the instructor only, in 
the MFK students share the responsibility of motivating others to comment 
and motivating others to have a voice. Because each person's comment is 
made public to peers in the course, students who may be insecure or lacking 
in confidence are likely to become more comfortable in engineering courses 
when they find that the concerns that they have are shared by others. Group 
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assessment using MFK provides an immediate opportunity for ideas to be 
debated among the community of students and allows instructors to weigh 
the comments of students in light of how they react to each other's 
suggestions. 

Table 1. Themes and Sample Questions for an MFK Evaluation 



A. Relevance to Engineering Practice - students perception about importance 
of current education and impact on future employment and success in 
engineering, open-ended problem solving, application of theory, analyses, 
technical excellence, social and communication skills, teamwork 

B. Learning Experiences - activities in class are relevant to learning, 
comparing synthesis courses with non-synthesis courses, learning 
environment, hands-on, access to new technology, multiple delivery formats, 
multidisciplinary exposure 

C. Class Atmosphere - students' opinions about class environment, ability to 
find help, camaraderie, comfort with gender and ethnic issues, receptiveness 
to learning styles, supportive/hostile environment for engineering success 

D. Open-ended Problem Solving/Design - opportunities for hands-on design, 
creativity, learning about concurrent engineering, design for X, solving 
original problems best way to learn design, opportunity for creativity, 
concepts applicable to new problems, concurrent engineering, DFX, linking 
coursework to design tasks 



Another feature of group assessment is that it is interactive. If a student raises 
a point which is not well understood by the instructor, the instructor (or 
another student) can probe further, asking for clarification or elaboration. 
Likewise, if a student critiques or questions some aspect of the course, the 
instructor is free to explain his or her rationale for the course design. If new 
ideas are introduced, all have the opportunity to discuss them; compare this 
to, for instance, a course evaluation survey in which a student response to 
one item might suggest additional items for the survey. 

Assessment in a group also allows individual students to define him /herself 
with respect to others to establish a role. The nature of discussion is to take 
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positions, and students who agree or disagree with each other will naturally 
highlight these similarities and differences in their conversations. 

Why On-line Assessment? 

The Multimedia Forum Kiosk allows comments to be collected on-line: 
automatically and electronically. The advantage of creating an electronic 
record of all comments and interactions made by users is that data are easy to 
collect. Moreover, instructors can receive immediate feedback by reading 
students' responses interactively or review the feedback by reading 
comments compiled into a text file. This allows instructors to modify their 
courses and better tailor them to students' needs during instruction, rather 
than in the following semester as is typical of feedback from course 
evaluation surveys. 

Although MFK creates a permanent electronic record of all comments 
entered along with the time and date of their creation, students who revise 
their thinking on a topic can update their overview comments at any time. 
This feature captures both the process by which students think about different 
issues as well as their resulting knowledge of the issue. 

Another advantage of on-line assessment is that every individual can 
participate in the discussion at their own pace which is often not possible in 
panel review or face-to-face discussion. We have had great success in bringing 
cultural and linguistic minorities into the conversation because they may 
read, write, and revise contributions at their own pace. The asynchronous 
nature of a Kiosk discussion allows individuals to participate in a many-to- 
many discussion, but enter the discussion as one-to-many. In other words, 
each student has the time and space to have a voice. Additionally, 
individuals can choose to make anonymous comments, allowing less vocal 
and less confident students a role in the discussion. 

Asynchronous discussion also provides time for individual reflection. For 
instance, these representative comments demonstrate the quality of 
comments that MFK is able to capture among university students. 
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**Opinion Area: Designing & generating concepts in groups: help or hindrance? 

It really depends on the members of the brain storming group. If you have a 
very dominant team member in the brain storming group shooting down 
people's ideas, causing fixations and preventing context switch, concept 
generating in a group can be a disaster. As far as I am concerned, they (in a 
group) did not generate more brilliant ideas than my individual effort. 

Designing in groups may be a hindrance for those who are afraid of being 
ridiculed or judged. I found that the best teams, whether design or whatever, 
work well because they have broken down barriers between the members. An 
understanding and respect for individuality exists between the members and 
they can work effectively together. 



Before working together as a group you have to learn how to cooperate, how to 
work creatively. The design fixation experiment, and the reports are a very 
good way to learn team work from one's own mistakes and the mistakes of the 
others. The best way to create ideas would be, that at first each team member 
should think about the problem for his own. At the next day there should be a 
meeting where the ideas are discussed. 

** Opinion Area: What are some effective ways to teach design? 

I think that effective teaching for design or any other subject should consist of 
using many teaching methods such as hands on experience (industry 
experience), lecture, discussion, reading, etc. Because students have different 
preferred learning styles (i.e. some learn best by experimenting, some by 
reading), a multitude of teaching methods will help all the students learn. 

It occurred to me that engineering programs, unlike those in fine arts or 
architecture, do not have gut- wrenching critiques and juries. These are highly 
instructional and formative activities necessary in the teaching of these 
disciplines. Would this approach be helpful in engineering? 

**Discussion Area: What are some effective ways to teach design? 
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It seems to me that case studies cannot really be said to convey the design 
process as it happens very well. They give the highlights, and these can be 
learned from, but learning how something was done is not the same as doing 
that thing. 

True. Maybe a deeper evaluation of a design using the case study as a guide 
would be more helpful than just reading and discussing the case study itself. 
Adding hands-on experience to the case study. 

One learns by doing. As Tod said, the ability to design requires practice at 
designing. You can't learn how to juggle, or drive a car, or ride a bike by 
simply watching another perform it. As with any other skill, design requires 
practice. 

But without some background, the first few months you will be groping 
without any direction. 

It would also be useful to see examples of bad design. Especially if one didn't 
know what the good/bad aspects of a design were until after the analysis had 
been performed. 



Why Multimedia/Hypermedia Assessment? 

Multimedia features were used in two different ways on the Kiosk. First, face 
icons of all participants of the discussion made identities salient. This allowed 
discussion to take on a more personal tone since participants were reminded 
that real people were behind each comment. A second feature of multimedia 
was the use of images, computerized documents, and digital movies as 
discussion stimulus materials. Multimedia (especially video) was used 
specifically to bring engineering examples into the discussion, to link 
examples of curricula students had been experiencing in the course, or to 
import personal stories and add a more human feel to the topic. We included 
interviews with engineers from industry, replayable simulations, and 
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interviews with engineering students at other campuses discussing the same 
issues. These images and videos were often more poignant and more 
effective at starting discussion than vague, depersonalized references in text. 



Analysis: purposes and strategies 

As mentioned above, the MFK system was intended to fill gaps left by other 
methodologies. In the courses we worked with, other assessment strategies 
included student surveys, grades and tests, portfolios, and analysis of 
videotapes of student activities. Our method succeeded at providing 
effective, formative evaluation of curricular changes to the instructors. The 
methodology proved to be of much lower cost than detailed methodologies 
such as video interaction analysis (Jordan and Henderson, 1995), and more 
informative than simple methods such as student evaluation surveys. It also 
proved to be able to capture non-traditional outcomes such as attitudes and 
strategies for collaborative work where traditional assessments such as tests, 
grades, and even portfolios sometimes failed. 

Our primary goals in our analyses were to 1) offer informal, timely feedback 
to course instructors, 2) to capture information related to our predetermined 
themes, and 3) to identify good and bad aspects of the curricular reform that 
were not captured in our themes. We hoped to provide rich examples that 
explained and supported other methodologies. 

Analysis with respect to the first goal, timely feedback, was accomplished by 
having instructors participate in discussions and summarize issues raised in 
the Kiosk during their lecture time. Summaries of the discussions were also 
provided to the instructors, who would often share these with instructors of 
other courses. At least one instructor made major changes in the course 
outline for their course midstream due to feedback received through the 
Kiosk. 

Analysis with respect to the second goal, information on specific themes, was 
achieved by classifying each comment in every topic according to which of the 
themes they addressed. Within themes, categories were created for types of 
responses. For instance, in the "learning experiences" theme comments 
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were categorized according to the specific learning method discussed 
(multimedia presentations, case studies, design projects, etc.) and the 
student's response to the curriculum in that area, where possible. In addition, 
some analysis of the discourse processes proved illuminating for our themes 
as well. For instance, we tallied comments-per-person ratios and proportions 
of comments made anonymously in order to gain insight into "classroom 
atmosphere". We found large differences between classrooms, with 
instructors who were stereotyped as unapproachable or harsh generating large 
proportions of anonymous comments. Interestingly, anonymous comments 
and critical comments were not one and the same, with most students 
signing their criticisms in certain courses, while many students made even 
positive comments anonymously in other courses, (see Figure 4) This may 
support the use of anonymity as a measure of comfort in the classroom 
atmosphere. 

Our third goal of identifying ideas outside of our planned analysis was also a 
success. In summarizing the discussions for the instructors, we identified 
issues which had been raised but that were not captured in our themes. 
Students felt comfortable raising these ideas. For instance, in one course 
students raised points about the organization of their course. Even though 
none of our topics specifically addressed course organization, many students 
took the opportunity the Kiosk presented to complain about these problems. 
These same issues were raised in course evaluations later on. Other students 
commented on topics ranging from the character of their particular college 
town to the problems faced by non-native English speakers at a university. 

We did preliminary cross-validation of the Kiosk method by comparing the 
summary reports generated with course evaluations performed on a 
standardized survey. Where non-thematic issues were raised in the MFK and 
incorporated into the summary, we checked against the standardized survey 
to see if any of the questions applied to that issue. Where they did, we 
examined the survey results for that question in that particular course. In 
every case, results for the course differed from the baseline in the way 
predicted by the Kiosk summary. 
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Currently, work is underway to use the structure of Kiosk discussions to 
measure scientific inquiry and science conceptions in a middle school science 
course. We believe that the Kiosk will continue to provide rich data sources 
for various types of analyses and assessments, from the informal and casual 
to the highly rigorous. 

Conclusions 

In summary, computer discussion tools can provide a useful platform on 
which to base assessments of curricular changes. In our case, the Multimedia 
Forum Kiosk proved especially fruitful in examining curricular reform, 
where goals were vague and not based on declarative knowledge. By 
involving the instructors in conversations with their students, the impact of 
curricular changes on students was readily felt and fed the process of reform. 
Capturing discussions is a viable way to examine processes and attitudes, and 
electronic discussion tools with multimedia fill an important niche in open- 
ended assessment methodologies. 
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Figure 4: Comparison of anonymous vs. named comments in MFK 
evaluation of different engineering courses 
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THE CATHOLIC UNIVERSITY OF AMERICA 

Department of Education, O’Boyle Hall 
Washington, DC 20064 
202 319-5120 



March 1995 

Dear AERA Presenter, 

Congratualations on being a presenter at AERA. The ERIC Clearinghouse on Assessment 
and Evaluation would like you to contribute to ERIC by providing us with a written copy of 
your presentation. Submitting your paper to ERIC ensures a wider audience by making it 
available to members of the education community who could not attend the session or this 
year’s conference. 

Abstracts of papers that are accepted by ERIC appear in RIE and are announced to over 
5,000 organizations. The inclusion of your work makes it readily available to other 
researchers, provides a permanent archive, and enhances the quality of RIE. Your 
contribution will be accessible through the printed and electronic versions of RIE, through 
the microfiche collections that are housed at libraries around the country and the world, and 
through the ERIC Document Reproduction Service. 

We are gathering all the papers from the AERA Conference. We will route your paper to the 
appropriate clearinghouse and you will be notified if your paper meets ERIC’s criteria. 
Documents are reviewed for contribution to education, timeliness, relevance, methodology, 
effectiveness of presentation, and reproduction quality. 

To disseminate your work through ERIC, you need to sign the reproduction release form on 
the back of this letter and include it with two copies of your paper. You can drop of the 
copies of your paper and reproduction release form at the ERIC booth (615) or mail to our 
attention at the address below. Please feel free to copy the form for future or additional 
submissions. 

Mail to: AERA 1995/ERIC Acquisitions 

The Catholic University of America 
O’Boyle Hall, Room 210 
Washington, DC 20064 




Lawrence M. Rudner, Ph.D. 
Director, ERIC/AE 




Clearinghouse on Assessment and Evaluation 



